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Abstract 

This paper presents a holistic framework for evaluating learning materials and designs for 
learning. A holistic evaluation comprises investigations of the potential learning potential, 
the actualised learning potential, and the actual learning. Each aspect is explained and 
exemplified through theoretical models and definitions. 
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Introduction 

The aim of this paper is to present a holistic framework for evaluating learning materials and designs for 
learning. We and our colleagues have been participating in the development of a variety of learning materials 
based on empirical research and theoretically generated design principles. On the basis of both empirical data 
and theoretical principles, we have argued for the success of some learning materials and the shortcomings of 
others (Bundsgaard, 2005, 2009, 2010a, 2010b; Bundsgaard & Hansen, 2010; Hansen, 2010a, 2010b, 2011). But 
we have felt a still stronger need for a firmer basis for these claims and a still more urgent call for a thorough 
framework on the basis of which comparisons between learning materials can be made. The ultimate, yet 
unattainable, goal is to develop a yardstick to measure which one of several learning materials is the best. 

Before we present the framework, we will give a short overview of essential work done in the area. A number of 
studies and frameworks focus on an analysis strategy where the learning material or course is evaluated as a 
text, i.e., the object of study is the explicit and implicit descriptions of student behaviour and expected learning 
outcome. Baker (2003) has developed a framework for design and evaluation of Internet-based distance learning 
courses on the basis of Ralph Tyler’s (1949) theory of curriculum development, Bloom’s (1956) learning 
taxonomy and Blanchard and Johnson’s (1984) SMART framework. Leacook and Nesbit (2007) have 
developed a very popular and well-described framework called Learning Object Review Instrument (LORI). By 
using LORI, evaluators can create reviews consisting of ratings and comments on nine dimensions: content 
quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, 
accessibility, reusability, and standards compliance. A primary goal of LORI is to balance assessment validity 
with efficiency of the evaluation process (Leacock & Nesbit 2007), and thus evaluation is not based on close 
reading or detailed analysis, but on the evaluator’s overall impression of the learning object’s dimensions of 
quality. 
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Another congenial framework is Lceremiddeltjek (English: learning material check out), a systematic tool for 
assessing learning material through evaluation within six parameters: accessibility, progression, differentiation, 
facilitation, coherence and legitimacy. The framework is intended to be used by researchers as well as teachers 
to give a swift evaluation of learning material, qualified by a theory-based model which offers a functionalistic 
approach to learning materials as tools helping the user with solving specific problems related to learning and 
teaching (Hansen & Skovmand, 2011). 

Another group of frameworks focus on teachers’ evaluation of learning materials. In the Mod4L project 
(Falconer et al., 2007), the goal was to investigate what made teachers choose a specific learning design, and to 
develop a framework to evaluate learning designs from that perspective. The framework consists of four stages 
of sharing and reuse of learning designs and of eight principles related to the descriptions of learning designs 
and their underpinning pedagogies that influence practitioners choice of learning designs (Falconer et al., 2007). 

A third group of studies focus on evaluating learning material in concrete situations through ethnographic 
research, and, in some cases, such studies include test-based evaluation of the participants’ learning outcome. 
Harley, Seals and Rossen (1998) studied a learning tool to support object-oriented design of computer programs 
through a problem-based approach. They used video to capture the students’ collaboration and screen-capture to 
record the interactions with the computer. The analysis focused on the quality of the learning material through 
analysis of students’ interaction with the system and the resulting success in problem solving. Further, the 
research group centered around Neil Mercer and Rupert Wegerif have developed a number of innovative 
computer programs based on the hypothesis that students improve their thinking skills by developing a few 
ground rules of dialog. The group has done a number of evaluation studies to support their hypothesis including 
discourse analysis of transcripts based on video recordings of students’ collaboration, pre- and post-intervention 
reasoning tests etc. (Wegerif 2004; Wegerif & Mercer 2000). In Krauss and Ally (2005), the outcome of 
integrating the evaluation of learning objects in the design process is studied through a variety of methods from 
student and faculty ratings and survey questionnaires to think-aloud sessions. 

The literature reviewed here and all other literature we have knowledge of in this area focus on some aspect of 
learning materials, such as its intentions, its use, and the learning outcome, but none provide a holistic view of 
the task of evaluating learning materials. Therefore it is our intention to provide such a holistic perspective. 


Holistic evaluation of learning materials 

As will appear from the following, we use two connected but separate concepts: learning materials and design 
for learning. We understand learning materials as artifacts, e.g. textbooks, blackboards, computers, while we 
define a design for learning thus: 

Definition: A design for learning is a constellation of artifacts (which can be called learning materials) 
arranged (in space) and articulated (in time) by someone with the intention to initiate and support 
someone’s learning. 

This means that a design for learning can comprise several learning materials and learning materials can 
“prescribe” or implicitly presuppose a design for learning. A prototypical learning material can, for example, be 
produced by a group of authors and published by a publisher as a text book or an ICT learning object, while a 
design for learning can be the product of a teacher’s explicit or implicit plans for the layout of the room with the 
texts and objects it contains. Learning material includes more or less explicit descriptions or expectations of the 
learning situation. An ordinary textbook implicitly presupposes that the teacher will teach from a desk at the 
front of the room while the students will sit in rows and listen, and carry out the assigned tasks. For example, an 
educational computer game presupposes that the students will sit in front of their computers and play while a 
storyline framework presupposes that teachers and students will collaborate in many places and through a 
variety of interactions. 

On the basis of these definitions, we present the framework as a three-step method for performing a holistic 
evaluation. 

Definition: A holistic evaluation of learning material comprises investigations of: 
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a. the potential learning potential, that is, the affordances and challenges of the learning 
material, and the competences supposedly supported when working with the material; 

b. the actualised learning potential, that is, the potential for learning when the design for 
learning is enacted by integrating the learning material in a situation in a given context; and, 

c. the actual learning, that is, how the participants actually develop their competences through 
working with the learning material or enacting a design for learning. 

The first part of an evaluation consists of analysis and interpretation of the learning material as a text. This 
analysis leads to a description of the potential learning potential. When introducing learning materials into a 
specific situation, some aspects of the potential of the learning material are actualised through integration with 
the historical constellation of artifacts in the situation. The students and teachers more or less deliberately 
choose to supplement the learning material with tools and materials, and these phenomena comprise the design 
for learning, which constitutes the actual learning potential. When the teachers and students have worked with 
the learning material and have thereby enacted the design for learning for a period of time, they are expected to 
have learned something. The producers of the learning material might have articulated the expected outcome, 
and the analysis of the potential and actual learning potentials may point to a further range of potential 
outcomes. The actual learning outcome can be measured against these two forecasts. 

In all probability, no single evaluation of learning materials can comprise all aspects of this complex framework. 
However, the framework can be used as a heuristic to evaluate and discuss the shortcomings and benefits of 
evaluations of learning materials and as a tool for planning an evaluation of a learning material. 

The triple division can be understood as a temporal structure: before, in, and after use. However, it is also 
important to understand it as a methodological structure. Each of the three points can be regarded as a 
perspective on the evaluation of learning materials, each characterised by its own logic and accessibility because 
the three temporal phases correspond to three different types of participation and relation to the teaching 
situation. 


Some basic definitions 

The object for evaluation seems quite distinct and obvious, at least in the case of learning materials. But what 
are learning materials? At first glance they appear solid and observable; objects that can be investigated, with 
distinct features to describe. But in fact the notion of learning materials is a fluid concept used to denote 
different types of materials and resources in relation to teaching situations: textbooks, on-line learning materials, 
educational games, mobile phones, blackboards, digital boards, charts and literature among others. For the 
purposes of our study, we define learning materials by dividing them into three types: 

Definition: Types of learning materials 

• Functional learning materials (tools) characterised by their facilitation of learning and teaching-. 
including black and white boards, computer applications, projectors, and mobile phones. 

• Semantic learning materials (texts) characterised by their meaning as constituted by signs and 
semantic references', including film, literature, charts, pictures, paintings and other texts and objects 
with references to specific domains of experience. 

• “Didaeticized” learning materials characterised by combining tools and texts and facilitating 
learning and teaching-, including textbooks, online teaching materials, and educational games. 

The word didaeticized is a neologism created from the word “didactic,” which in its Germanic form ( Didaktik ) 
signifies planning and reflecting on teaching. Thus, something which is didaeticized is the result of the process 
of anticipating and prescribing the activities that the teacher and students (should) perform; that is, a 
didaeticized learning material has a more or less implicit design for learning inscribed in it (Hansen, 2006). 
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The first two types have something in common, because they do not have designs for learning coded into the 
material; they become learning materials through being part of a contextual design for learning which co¬ 
determines the use of the learning material in the situation. These different types of learning materials and their 
integration into designs for learning must be taken into account in order to have an inclusive definition of 
learning materials. Learning materials are materials and tools integrated into a design for learning: whether a 
contextual design for learning co-determining the use of the materials, or an immanent design for learning coded 
into the material, or both. Thus, as we have already underlined, there is a close relationship between the notions 
of learning materials and designs for learning, at least viewed from the material point of view. 

In the definition of the concept “design for learning,” the word “artifact” was included. We use this concept in a 
similar way to theoreticians (for example. Cole, 1996; Saljo, 2005; Wartofsky, 1973), who define artifacts as 
both material tools and objects and as ideal “in that their material form has been shaped by their participation in 
the interactions of which they were previously a part and which they mediate in the present” (Cole, 1996, p. 

117). Some artifacts have a material aspect while others are more or less completely ideal even though ideas or 
mental models need some kind of external representations (for example, spoken words) if they are to be shared 
with others. In an educational context, the external representation is crucial, underpinning the important role of 
learning materials. We have elaborated a typology of artifacts adapted from Cole (1996), who, in turn, adapted it 
from Wartofsky (1973), while emphasising that there are important differences between our definition and those 
of Cole and Wartofsky. 

Definition: Artifacts 

• Primary artifacts: perceptual objects we can point to, such as (a) physical objects, tools and 
technologies, (b) layout of the physical environment, and (c) texts, that is, the physical signs or 
marks that can be pointed to; 

• Secondary artifacts: conceptual objects we can refer to as existing, such as (a) rules: laws of 
nature, laws of the state, moral laws, and (computer) algorithms, (b) processes: social 
algorithms: what is to be done first, next, and then in a social context, and (c) mental and social 
models: world view, personal relations, authority, ideology; and, 

• Tertiary artifacts: objects of the imagination that we can refer to as if they were existing, 
which Wartofsky (1973) defined as imagined worlds, like those of art and fiction. 

“Artifact” is a term used for all kinds of objects and processes that are the results of human activity. In our 
conception, artifacts are social phenomena that derive their value and meaning from the situational and social 
context in which they are used and created. This becomes clear when we analyse concrete use of learning 
materials in situations where (a) the status of artifacts is constantly negotiated, and (b) interactions between 
teachers and students are mediated by artifacts. 


Potential learning potential 

There are some consequences and noticeable challenges emerging from our definitions and hence some 
challenges for the demarcation of the analytical field that is the object of evaluation of learning materials. First 
of all, we will point out that although desk-based analysis and evaluation of learning materials might result in 
relevant and important findings, they will suffer from being de-contextualised and distant from their situated 
use. It is possible to a certain degree to analyse and evaluate learning materials without leaving one’s desk, but 
one must be aware that the analysis is (to be) grounded on a presupposed knowledge of prototypical designs for 
learning in prototypical situations. It is not enough to analyse and evaluate the learning material on the basis of a 
“check list” or a formalistic grammar of the well-formed text. We therefore present a functionalistic and 
phenomenological model for analysing designs for learning. This model (Figure 1) describes a number of focal 
points or perspectives on learning materials and identifies a series of steps constituting a meticulous textual 
analysis of the specific learning material. 


2011 Vol. 4 No. 4 


34 



Bundsgaard & Hansen 


ULD 


J ournai of Learning Design 


I.Ccnlmtt 


Ehuai 

Expaftuion 


) 


5, P«f*pfrrtlve 

"Ri «roiaaZMiafi ' 

i Ttl«*>en fr? »p*=ilV J 
paruap&nc* 


2 . Ch arac!e ri e. lies 



3. Annlysi 


5 


InlerpretJlion 



/ 


li^rpiann 


ActMLy 

Madiuir, ■£: HkS^ tv 


Fruvnng 

CcnLanE 

\ Focus yr 

Wcrfc: "ji mi", hod ■ 

Ganr# and wxttypM 

Teaching nwtbcli 


lni#nnpn-Miiv 

isd MntMl 

For-* of kr.prfladga 



Figure 1. Textual evaluation of learning materials 


Context and characteristics are the initial descriptive steps focusing on the context of the learning material and 
describing the first impression of the design, understood as the appearance of an intentional wholeness 
encompassing both form and content. Analysis is the pivotal point in the textual evaluation, it has three aspects 
of equal importance, that is, having the status of interdependent parts of a structural whole. We outline this 
phase further below. Interpretation and Perspective are the final interpretative steps towards a textual 
evaluation, articulating the potential learning potential. 

Each step in this model requires further explanation and elaboration, in this context, we focus on the analytical 
triangle shown in Figure 1, which we explain with reference to the speech act, that is, when someone is saying 
something, about something, or in order to have someone do something. All learning materials and designs for 
learning can be seen and analysed as speech acts (cf. Austin, 1962) containing a perceptual representation (the 
expression: in Austin’s words, the locutionary act) signifying a thematic field of attention (the intention: in 
Austin’s words, the illocutionary act) and appealing to an addressee, framing some kind of enactment (the 
activity: in Austin’s words, the perlocutionary act). 

Even in traditional teaching, where the teacher lectures and the students sit in rows, there is an act and a framing 
of a certain activity, because the students are supposed to sit still, listen and understand the theme of the lesson 
in a certain way. Thus the teacher’s speech contains an implicit speech act: “I want you to sit still, listen very 
carefully, see this before your eyes and understand it in a certain way, specified by virtue of my framing.” This 
triangle becomes explicit when the participants experience problems with the communication and the teacher or 
students have to focus on one of the three aspects; for example: 

1. expression, when the forms of representation (medium, modes, genres and text types) must be modified 
in order to achieve an adequate presentation; 

2. intentionality, when there is a need for reorganizing the intentional structure in order to link the aim 
and content to the students’ life world, and finally, 

3. activity, when the teacher finds it necessary to make an appeal to the students or request a certain 
action. 

Similarly, the triangular structure often becomes explicit and observable when designs for learning are 
coded into learning materials. We use the analytical triangle as a frame of reference when analysing the 
learning materials. Our focal points can be divided into four categories: 
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A. Students’ perspective: accessibility, differentiation, progression, and organisation of teaching 

B. Teacher’s perspective: facilitation (while planning, implementing and evaluating teaching), and 
integrity (concerning immanent coherence within the material itself and transcendent coherence in 
relation to the norms and values of the teacher). 

C. Situational perspective: organisation (of interactions, space and time, room requirements, 
organization of furniture, consecutive time needed etc.), and relations (between participants, and to the 
outside world). 

D. Society’s perspective: integrity (concerning transcendent coherence in relation to the norms and values 
of the actual school culture), and legitimacy (measured using governmental policy, national curriculum 
and up-to-date knowledge as yardsticks). 

It is possible to argue that these focal points are core elements in teaching situations, with their basis in general 
theories of learning and education. In this connection, we restrict ourselves to arguing for their relevance with 
reference to the logic of the teaching situation: a) the students are supposed to learn something and develop 
relevant competences within the horizon of a democratic society, b) the teacher is supposed to carry through the 
planning, implementation and evaluation of teaching, c) the participants are rooted in a situation in time and 
space (and thereby confined by the material basis) and co-determined by the relations between participants and 
their historically developed local culture; and d) the realisation of learning and teaching aims is supposed to be 
legitimated with reference to the surrounding world. These shifts in focus from the students’ to society’s 
perspective imply an epistemological and methodological shift from cognitive and socio-semiotic to 
sociological approaches to the teaching situation. 

We use the analytical triangulation of expression, intentionality and activity as a pragmatic tool to ensure an 
adequate frame for analysing the focal points and mapping the learning potential, stressing that there is no direct 
access to learning and the content of teaching, whereas all processes of learning have expression, that is, 
appearance of forms and representations, as their points of departure. 


Competences 

The potential learning potential can be described in terms of the competences that students can be expected to 
develop. These can be described from two perspectives. Firstly, competences are closely related to the situation 
in which they are developed and used and can thus be described as competences of the community of practice 
(Wenger, 1998) in which the students participate. Thus students first learn to act as students in the ways in 
which the institution, design for learning and other people structure their participation. If students are expected 
and forced to sit in silence and listen, that is what they learn - though they might of course also learn something 
through what they hear and see. If, on the other hand, students participate in a more elaborate practice, for 
example as journalists on a newspaper, they can be expected to learn to act, think and feel like journalists 
(Bundsgaard, 2009; Shaffer, 2006). From a community of practice perspective, competences can be seen as 
knowledge of and capacity for the practices, attitudes and approaches of a given practice. The potential learning 
potential can thus be outlined on the basis of a description of the practices in which students will participate 
while enacting the design for learning co-determined by the learning material under scrutiny. 

Secondly, competences can be described on a more generalised level (cf. Bundsgaard, 2006; Rychen & 

Salganik, 2003) as the cognitive and practical skills, the knowledge, attitude, and motivation that students 
develop while using the learning material, cf. the demand-oriented competence definition by Rychen & Salganik 
(2003). This perspective defines competence as the capacity to rise to the challenges of a situation; in that way it 
is comparable to the first perspective, but will have more focus on transferability between practices, and will 
seek to describe competences in a cross-situational wording. For example, when students collaborate on the 
production of a newspaper, this perspective will focus on the students’ development of productive 
communicative competences and describe these under such headings as attention to the addressee, knowledge of 
the typical characteristics of the genre and ability to conform or consciously deviate from it. 

To estimate the potential learning potential, it is necessary to have a thorough understanding of the prototypical 
design for learning implied by the learning material, and thus to analyse the design in all its complexity. This 
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part of the evaluation is a textual analysis and an analysis of the implicit or explicit description of the teaching 
and learning situation. 

The next step in a holistic evaluation of learning materials is research into the actual use in practice. Such 
investigation can lead to an identification of the actualised learning potential. 


Actualised learning potential 

The actualised learning potential of learning materials is a function of several factors in the situation where the 
learning material is used. Thus, a design for learning unfolds some potentials of the learning material in the 
situation and prevents others from unfolding. Other important factors co-determining the actual learning 
potentials are what could be termed the Zone of Proximal Development (see, Vygotksy, 1997/1931) of the 
participants (students and teachers) and the organisation of the situation (participants, space and time, history, 
context, and cultures). It is therefore clear that what is observed in a concrete situation is not solely due to the 
learning material. 

To further develop the understanding of how learning material functions together with other factors in the 
teaching situation, we elaborate on three dimensions: the spatial situation, the social relations, and the temporal 
processes. 


Spatial situation and social relations 

One of the most common conceptual models of the teaching situation is the so-called learning triangle, which 
depicts three important aspects of a teaching and learning situation, namely the teacher (T), the student (S) and 
the content (C) being taught. But the learning triangle has a number of shortcomings, especially if it is the 
mental model of the participants or the producers of the learning materials. It conceals some of the most 
important traits of the teaching and learning situation: for instance, that there is more than one student, that 
artifacts are dealt with by participants who are related in complex power relations, and that the interaction is 
situated in a context. These shortcomings are addressed in the model of the teaching and learning situation in 
Figure 2 (cf. Bang & Door, 2007; Bundsgaard, 2005). 



Context 


M: Medium 
M: Mark 
T: Text 

C: Communication 
technology 

|: possess|possessed 
A: Artifact 
T: Teacher 
S: Student 
S :Third subjects 


Figure 2. Teaching situation 


A teaching and learning situation is a communication situation, that is, it is made up of persons or groups of 
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persons communicating by means of communication technologies (C) to position and consume marks (M) in 
media (M) as texts (T). The media may be paper, screen, or stone while the mark might, respectively, be the ink- 
stains, light-dots (pixels), or carving made by using a technology, perhaps, a printer, a light emitter, or a 
hammer. 

There are always at least three subjects participating in a communication situation: SI and S2, who produce and 
consume the marks in the media, and S3, those who participate in setting the stage, existing in person or in the 
minds of SI and S2 or experiencing the consequences of the interaction between SI and S2. S3 can be 
understood as the ones influenced by or influencing the situation. Because we are addressing questions 
regarding teaching, we substitute the general concepts of SI and S2 with the specific roles of teacher (T) and 
student (S). 

This model shows that there is more than one student in a teaching situation and that the students relate to each 
other as well as to the teacher in distinctive ways. Even when the students are not expected to work together, 
they will make a difference to each other. Through the concept of “possessed” texts (|T) the model moreover 
shows that each student and teacher has his or her own individual understanding of the situation and the texts 
and artifacts involved. In a teaching situation, the participants each carry their own understanding of the 
“meaning” of the manifested texts and other artifacts. Successful learning material therefore supports many 
different acquisitions of the content, and not least, it allows for the presence of several interpretations and value 
systems in the situation, leading to different equally well-grounded understandings and texts. Nothing happens 
outside of a context. We define the context as the entirety of artifacts, subjects, society and culture related to the 
situation. In principle, that consists of the whole world, but in a specific situation some part of the context will 
be more closely related to the situation than other parts. 

Temporal processes 

The models presented above are situational models representing still images of the situation. But as teaching and 
learning take place in time, an investigation of processes is needed in order to understand how learning materials 
prescribe and support these processes. We present a few models showing how different perspectives lead to 
different insights. 

• Interaction sequence models 

Following studies into classroom talk patterns, Sinclair and Coultard (1975) described a typical teaching process 
as consisting of three functions: the teacher initiates the process by asking a question, a student responds, and 
the teacher evaluates the response. Sinclair and Coultard (1975) briefly describe this as the IRE structure of 
teaching. Extending from this line of thought, other models of teaching and learning can be described as a 
number of speech acts. For example, Wegerif and Mercer (2000) point out that introducing a single component, 
namely student discussion of the question before responding, may lead to a completely different process and 
learning. This type of interactional process model is open to ethnographic observation supported by video 
recordings, and the function might for example be found through analysing transcribed passages of student talk 
and action. 

• Workflow models 

On another level, we find workflow models showing how teaching and learning are comprised of sequences of 
interaction sequences, that is, the participants enact a certain workflow. Staffan Selander and his group of 
researchers are developing and working with such a model, which they call the learning design sequence. This 
model (Figure 3) shows how teaching and learning are performed as transformations of content. In the primary 
transformation unit, students transform the modes and media (which we understand to be what we call learning 
materials) by working with the content in a social context, thereby forming their understanding. In the secondary 
transformation unit, the students transform their forming to a representation, which in itself or through 
presentation to an audience gives them the opportunity to discuss and meta-reflect on their learning process. 
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Formal - LEARNING DESIGN SEQUENCE 


Primary transformation unit 

interest and social interaction 


Secondary transformation unit 


Learning resources 


Curriculum setting > transforming 


Institutional 

norms 


teacher s interventions 

formative assessment 



Figure 3. Learning design sequence (Selander, 2008, p. 17) 


Even though this is described as a model of “teaching in formal contexts,” we will maintain that it is a model of 
certain kinds of teaching organisation where the focus is on processing knowledge, that is, for example, where 
students alone, in groups or together with the teacher in class investigate a subject, solve a problem or write an 
assignment. The model does not, at one end of the scale, describe a teacher giving lectures, nor, at the other end 
of the scale, does it describe situations where students collaborate on common tasks, for example simulating an 
out-of-school practice or producing art or handicraft products. 

Phase models describing the different phases or functions performed by students in activities such as project- 
based learning and simulations of out-of-school practices are complex workflow models. An example from the 
other end of the spectrum is a traditional teacher-led workflow. The teacher gives a lecture, checks the students’ 
understanding (often using IRE-structured interaction), points out which pages in the text book should be 
studied and which tasks solved, and then the students work on the tasks alone or in peer groups. Finally, the 
teacher leads a class conference in which individual students present their solutions to the tasks. We call this 
type of workflow a Lecture-Recitation-Seatwork-Plenary session (LeReSeP) (Bundsgaard, 2010b; Bundsgaard 
& Hansen, 2010). 

It appears that different workflow models accentuate different aspects of the complex processes in classroom 
practices. Different workflows offer differing opportunities for the participants in the teaching and learning 
situation. When participating in a traditional teacher-led workflow with integrated IRE-structured interaction 
sequences, students become good at answering questions, but they do not practise such activities as discussion, 
idea generation, and problem identification. Therefore certain competences are trained and developed in some 
workflows and not in others. Learning materials relate to workflows in two ways. On the one hand, learning 
materials (more or less implicitly) prescribe workflows, and on the other, the learning materials support the 
teacher and students in the implementation of workflows. Therefore the workflow models prescribed in the 
learning material should be part of the analysis of the potential learning potential. In the investigation of the 
actualised learning potential, attention must be paid to how far the workflow follows the intended models, and 
how it deviates from these models, and models should be proposed that might better describe the actual practice. 

The learning materials prescribe and support workflow, but, of course, they are not actualised until they are put 
into practice. The processes are orchestrated in practice by students and teachers who use the resources in the 
learning material as well as other tools and experience from other processes (including especially, of course, 
those known from previous educational contexts). When investigating how learning materials support workflow, 
focus can be oriented towards the ways in which these re-organisations are possible. 

For the participants, some workflows are well known and comfortable while others might be experienced as 
boundless and confusing. The adequacy of the prescription and support of workflows in specific learning 
materials is therefore very context dependent. 
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Researching into situated designs for learning 

The models above are intended to serve as heuristic models for research on practices both inside and outside of 
the classroom. When evaluating the actualised learning potential of learning material, one is oriented towards 
how the constitution of the situation, that is, the contextualisation of the learning materials, takes place. The 
analysis should focus on (a) which aspects of the potential learning potentials are actualised; (b) whether they 
work in this context, in which ways is the situation a success, how far do the participants feel engaged in and 
motivated by the work; and (c) what are the challenges in this context of using these learning materials and 
thereby enacting a design for learning. 

The final goal of these analyses is to generalise analytically from the specific context to more general claims 
about the affordances and challenges the learning material will meet in a context of the kind examined. 

Research into situations is oriented towards effects, that is, what are the effects of changing these factors in this 
situation. The research often makes use of anthropological methods, but it can also be more quantitative, with 
questionnaires answered by the participants, collection of products, artifacts used and created in the work, and 
participants’ self evaluations. 


Actual learning 

The last perspective of a holistic evaluation of a design for learning is the analysis of the outcome of working 
with the design for learning under scrutiny. Outcome can be measured in many ways and be related to many 
factors. A measurement and estimate of the outcome must take its point of departure in the explicit learning 
goals and in the potential learning potential that was analytically found in the first phase of the evaluation. 

When evaluating the outcome of a given effort, one must have a clear definition of what kind of outcome counts 
as important. With respect to education, the outcome is learning, which we define very briefly as change in 
capacity’ (llleris, 2002). More specifically, learning can be defined thus: 

Definition: 

Learning is the movement from chaos/conflict/non-capability that takes place in response to 
combination and change (or because the intuitive understanding or capability matures) so that the person 
is capable of doing something he or she could not do before (in situations in which it coidd not be done 
before). Here, capable of means: can do (bodily and mentally), can express (communicate), can 
understand and combine (think), can evaluate (ethically, politically), can perceive (sensibility), can feel 
(emotion), and wants to (attitude). 

The outcome of education should be learning, but not any kind of learning. Thus one must differentiate between: 

1. Intended learning 

2. Unintended, but valuable learning 

3. Unintended and undesirable learning 

A design for learning that supports the intended learning goals but at the same time makes the students learn 
something very undesirable (for example, that there is no point in participating in developing the community for 
the sake of the common good) might be even worse than a design for learning that does not support the intended 
learning goals very well. Thus, even in an “objective” measurement of learning outcome there is a normative 
aspect. 

One of the significant challenges of assessing the actual student learning is related to connecting the potential 
learning potential of the learning material to the students’ change in capacity. In this framework, the SOLO 
taxonomy ( Structure of the Observed Learning Outcome ) developed by John Biggs and Kevin Collis (1982) can 
serve as an offset for this purpose. 


2011 Vol. 4 No. 4 


40 



Bundsgaard & Hansen 


ULD 


J ournai of Learning Design 


Biggs presents the SOLO-taxonomy as a model on his homepage 1 . It consists of five steps from: incompetence 
(prestructural) over one relevant aspect {unistructural), several relevant independent aspects ( multistructural ), 
integrated into a structure (relational), and ending with learning being generalised to a new domain ( extended 
abstract). Each step is described by sample verbs indicating levels of understanding and a coordinate system 
with competence on the y-axis indicates that higher steps corresponds to higher competence. 

Biggs (2011) describes the characteristic of the progression thus: 

As learning progresses, it becomes more complex. SOLO, which stands for the Structure of the Observed 
Learning Outcome, is a means of classifying learning outcomes in terms of their complexity, enabling us 
to assess students’ work in terms of its quality not of how many bits of this and of that they got right. At 
first we pick up only one or few aspects of the task (unistructural), then several aspects but they are 
unrelated (multistructural), then we learn how to integrate them into a whole (relational), and finally, we 
are able to generalised that whole to as yet untaught applications (extended abstract), (para. 1) 

Biggs’ description can give the impression that learning is taking place as a linear progression from the simple, 
unstructured, to the complex and abstract. From a situated sociocultural view this conception is too simple. 
Rather, we as humans face and perceive a coherent whole, before we identify simple elements. In this regard, 
we face the world as a complex relational whole (as in Biggs’ Level 4). From this point on, we develop our 
competences in coping with the challenges we face and, thereby, we begin to be able to identify relevant 
independent aspects and investigate the connection between aspects (Levels 2 and 3). On this basis, our 
understanding of the relational structure (Level 4) and our competence in coping with still more complex 
challenges might develop. And, finally, our competence in generalising and abstracting will be able to develop. 
This is why we recommend that the Stmcture of the Observed Learning Outcome model is regarded as a verbal 
model perceived as a spiral, depicting that progression happens on both complex and simple levels at the same 
time. 

The SOLO model can be used to analyse different degrees of complexity in the students’ understanding of the 
content and context of a learning material. Because this understanding will be observable in the students’ actions 
in relation to the learning material’s expression and activity, students’ competence levels will be evaluated 
through an analysis of their actions and products related to these aspects. This corresponds to the secondary 
transformation unit in Selander’s (2008) model, which designate this phase as summative assessment. 

If we combine a verbal version of the SOLO taxonomy with the Model of textual evaluation of learning 
materials (Figure 1), it appears that we can analyse and compare the degree of complexity in the learning 
material’s representation of content with the students’ acquisition of the content and competence in coping with 
new challenges and thereby to what extent the learning material’s potential learning potential has turned into 
actual learning. 

In an analysis, for instance, the learning material’s initiation of activities can be classified in different levels of 
competence, depending on if the activities demand understanding on unistructural, multistructural or relational 
levels. It rarely happens that a single activity demands an extended abstraction, but from a more comprehensive 
analysis of the activities initiated and content conveyed, it can be evaluated if the material does prepare the 
ground for students developing a more extended abstract understanding. 

The SOLO taxonomy has mostly been used in relation to Mathematics, but it can also be used in Social Studies 
and The Arts to bring attention to the point that progression is not only related to the cognitive steps evident in 
Bloom’s taxonomy (Bloom, 1956) but also encompass different degrees of structural complexity in relation to 
the content. 

The methods used for measuring learning outcomes are often standardised tests, but as it appears from the 
above, such tests might be poor at assessing the more complex aspects of the students’ understanding, and thus 
such assessments might miss the point that the learning material tries to make. There is a range of interesting 
work done in relation to developing more appropriate assessment frameworks from OECD’s PISA (OECD, 
2009) to IEA’s ICCS (Schulz et al., 2011) and ICILS (Fraillon & Ainley, n.d.) studies and projects such as 
ATC21S (ATC21S, n.d.). But assessment can also be done as part of the actual practice by identifying signs of 


1 www.johnbiggs.com.au/solo_graph.html, visited September 17, 2011. 
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learning, such as changed patterns of action and student products, for example, written assignments, slide 
shows, posters, documentary, portfolios, and learning logs. 


Conclusion 

In this paper, we have presented a holistic framework for the evaluation of learning materials. Very few actual 
evaluations will include all three steps, let alone all the aspects of each step. Thus the framework is neither a 
cookbook nor a demand for completeness or exhaustiveness, it is a heuristic to substantiate the design of 
evaluation and to discuss or evaluate accomplished evaluations. 

Compared with LORI and Lasremiddeltjek introduced earlier, it is obvious that the holistic evaluation 
framework presented in this paper does not offer swift and simple to use evaluation typologies. It is not meant to 
be either swift or simple. The aim is to present the complex causality of using learning materials implying a 
whole range of challenges to be dealt with in future research on learning materials. 

Teachers are not supposed to take all the aspects of the framework into account while designing learning and 
teaching. It would not be advisable. Rather, it is useful to view the framework as a kind of map, depicting the 
field of challenges and thereby mirroring the need for educating teachers with a range of professional 
competences. Simultaneously, we have to extend the research in learning materials in all three aspects and 
temporal phases, so we can help qualifying the teachers’ evaluation of potential learning potentials with a basis 
in evidence on actual learning and the complex causal relations between potential and actual learning. It is not 
enough to say what works. We have to investigate and demonstrate under which specific circumstances learning 
materials actually work in real settings. 


References 

ATC21S. (n.d.). Purpose. Retrieved from http://atc21s.org/index.php/about/purpose 

Austin, J. L. (1962). How to do things with words: The William James lectures delivered at Harvard University 
in 1955. Oxford: Clarendon. 

Baker, R. (2003): A framework for design and evaluation of internet-based distance learning courses - Phase 
one - Framework justification, design and evaluation. Online Journal of Distance Learning 
Administration, 2(6). Retrieved from http://www.westga.edu/~distance/ojdla/summer62/baker62.html 

Bang, J. C., Door, J., Steffensen, S.V., & Nash, J. (2007). Language, ecology’ and society. London: Continuum. 

Biggs, J. (2011). SOLO taxonomy. Retrieved from http://www.johnbiggs.com.au/solo_taxonomy.html 

Biggs, J., & Collis, K. (1982). Evaluating the quality of learning: The SOLO taxonomy. New York: Academic 
Press. 

Blanchard, K., & Johnson, S. (1984). The one minute manager. New York: Berkley. 

Bloom, B. S. (Ed.) (1956). Taxonomy of educational objectives, the classification of educational goals - 
HandbookI: Cognitive domain. New York: McKay. 

Bundsgaard, J. (2005). Bidrag til danskfagets it-didaktik. Odense, Denmark: Forlaget Ark. 

Bundsgaard, J. (2006). Noglekompetencer med bud til de humanistiske fagomrader. Cursiv, 7(1), 27-57. 

Bundsgaard, J. (2009). A practice scaffolding interactive platform. In C. O’Malley (Ed.), Computer supported 
collaborative learning practices (pp. 522-526). New Brunswick, NJ: International Society of the 
Learning Sciences (ISLS). 


2011 Vol. 4 No. 4 


42 



Bundsgaard & Hansen 


ULD 


J ourna! of Learning Design 


Bundsgaard, J. (2010a). Faglighed og digitale kcrcmidlcr i undervisningen. Danskpcedagogisk tidsskrift, 4( 10), 
15-23. 

Bundsgaard, J. (2010b). Supporting the processes of teaching and learning: How digital learning platforms 
support progressive teaching, European Conference on e-Learning, 9, 81-89. 

Bundsgaard, J., & Hansen, T.I. (2010). Processer i undervisningen: Om bmgerdreven innovation af digitale 
procesvaerktojer, Lceremiddeldidaktik, 4, pp. 18-27. 

Cole, M. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Harvard University Press. 

Engestrom, Y (1987). Learning by expanding: An activity-theoretical approach to developmental research. 
Retrieved from http://communication.ucsd.edu/MCA/Paper/Engestrom/expanding/toc.htm 

Falconer, L, Beetham, H., Oliver, R., Lockyer, L., & Littlejohn, A. (2007). Mod4L final report: Representing 
learning designs. Retrieved from http://www.jisc.ac.uk/publications/publications/mod41fmalreport.aspx 

Fraillon, J., & Ainley, J. (n.d.). The IEA international study of computer and information literacy (ICILS). 
Melbourne, Australia: Australian Council for Educational Research. Retrieved from 
http://icils2013.acer.edu.au/wp-content/uploads/examples/ICILS-Detailed-Project-Description.pdf 

Hansen, J. J. (2006). Mellem design og didaktik. (Unpublished doctoral dissertation). Syddansk University 
(University of Southern Denmark), Denmark. Retrieved from 
http://www.humaniora.sdu.dk/phd/dokumenter/filer/Afliandlinger-95.pdf 

Hansen, T. I. (2008), Lasremiddeldidaktik - hvad er det? Skitse til en almen lceremiddeldidaktik. I: Tidsskrift for 
Lceremiddeldidaktik, 7(1), 4-13. 

Hansen, T. I. (2010a). Den fenomenologiske tilgang. In S.V. Knudsen & B. Aamotsbakken (Ed.), Teoretiske 
tilncerminger tilpedagogiske tekster (pp. 52-71). Kristiansand, Denmark: Hoyskoleforlaget. 

Hansen, T. I. (2010b). Multimodalitet som analysekategori. Viden om Icesning 3, Nationalt Videncenter for 
Lassning. 

Hansen, T. I. (2011). Kognitiv litteraturdidaktik. Kobenhavn (Copenhagen), Denmark: Dansklasrerforeningen. 

Hansen, T. I., & Skovmand, K. (2011). Fcelles mdl ogmidler. Arhus, Denmark: Klim. 

Harley, H., Seals, C., & Rosson, M.B. (1998). A formative evaluation of scenario-based tools for the learning 
object-oriented design. Crossroads: ACM Student Magazine, 1-5. 

llleris, K. (2002). The three dimensions of learning. Copenhagen, Denmark: Roskilde University Press. 

Krauss, F., & Ally, M. (2005). A study of the design and evaluation of a learning object and implications for 
content development. Interdisciplinary ; Journal of Knowledge and Learning Objects, 1, 1-22. Retrieved 
from http://ijklo.org/Volumel/vlp001-022Krauss.pdf 

Leacock, T. L., & Nesbit, J. C. (2007). A framework for evaluating the quality of multimedia learning resources. 
Educational Technology’ & Society, 70(2), 44-59. 

Novak, J. D., & Canas, A. J. (2006): The theory underlying concept maps and how to construct and use them. 
Technical Report IHMC CmapTools. Retrieved from 

http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMaps.htm 

OECD (2009). PISA 2009 assessment framework - Key’ competencies in reading, mathematics and science. 

Paris: OECD. 


2011 Vol. 4 No. 4 


43 




eb Journa! of Learning Design 


Bundsgaard & Hansen 


Rychen, D. S., & Salganik, L. H. (Eds.) (2003). Key competencies for a successful life and a well-functioning 
society. Cambridge, MA: Hogrefe & Huber Publishers. 

Schulz, W., Ainley, J., Fraillon, J., Kerr, D., & Losito, B. (2011). / CCS 2009 international report: Civic 
knowledge, attitudes, and engagement among lower secondary’ school students in 38 countries. 
Amsterdam, The Netherlands: International Association for the Evaluation of Educational Achievement 
(IEA). 

Selander, S. (2008). Designs for learning - A theoretical perspective. Designs for learning, 7(1), 10-22. 

Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers & Education, 46 (3), 223-234. 

Sinclair, J., & Coulthard, M. (1975). Towards an analysis of discourse: The English used by teachers and 
pupils. London: Oxford University Press. 

Saljo, R. (2005). Larande ipraktiken - et sociokulturelltperspektiv. Stockholm, Sweden: Norstedts Akademiska 
Forlag. 

Tyler, R.W. (1949). Basic principles of curriculum and instruction. Chicago. IL: The University of Chicago 
Press. 

Vygotsky, L. S. (1997/1931). The collected works of L. S. Vygotsky: Vol. 4. The history of the development of 
higher mental functions (M. Hall, Trans.; R. W. Rieber, Ed.). New York: Plenum. 

Wartofsky, M. W. (1973). Perception, representation, and the forms of action: Toward an historical 

epistemology. In M. W. Wartofsky (Ed.), Models (pp. 188-210). Dordrecht, The Netherlands: D. Reidel. 

Wegerif, R. (2004). The role of educational software as a support for teaching and learning conversations. 
Computers and Education, 43{2), 179-191. 

Wegerif, R. & Mercer, N. (2000). Language for thinking: A study of children solving reasoning problems 

together. In H. Cowie & G. van der Aaalsvoort (Eds), Social Interaction in Learning and Instruction (pp. 
358-402). Amsterdam, The Netherlands: Pergamon. 

Wenger, E. (2008/1998). Communities of practice. Learning, meaning, and identity’. Cambridge, UK: 
Cambridge University Press. 


Notes 

An earlier version of this article has been published as a conference paper: 

Bundsgaard, J. & Hansen, T.I. (2011). Holistic evaluations of learning materials. In J.R. Rodriguez, M. 
Horsley, & S. Knudsen, (Ed.), Local, National and Transnational identities in Textbooks and 
Educational Media : Ten International Conference on Research on Textbooks and Educational Media 
September 2009 Santiago de Compostela - Spain. Santiago : IARTEM, 2011. 

In order to give an overview of the holistic evaluation framework presented in this paper, the authors have 
produced a concept map, which includes all the core concepts and shows how they are connected. This concept 
map can be used as a checklist when preparing an evaluation, and as an instrument to guide a discussion on 
aspects of learning materials that are encompassed by a given evaluation. The concept map can be found at 
www.laeremiddel.dk/holisticevaluation 
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